ON THE TRACE OF HECKE OPERATORS FOR 
MAASS FORMS FOR CONGRUENCE SUBGROUPS 



J. B. CONREY AND XiAN-JiN Ll 

I Abstract. Let Ex be a Hilbert space, whose elements are functions spanned by 

■ the eigenfunctions of the Laplace-Beltrami operator associated with an eigenvalue 

A > 0. The norm of elements in this space is given by the Petersson inner product. 
In this paper, the trace of Hecke operators T„ acting on the space Ex is computed 
for congruence subgroups ro(A'') of square free level, which may be considered as the 
analogue of the Eichler-Selberg trace formula [11] for non-holomorphic cusp forms of 
weight zero. 

h" 

:^ 

. 1. Introduction 

Let be a positive integer greater than one. Denote by ro(A^) the Hecke congru- 
ence subgroup of level A^. The Laplace-Beltrami operator A on the upper half-plane 
H is given by 



Let D be the fundamental domain of Fq ( A^) . Eigenfunctions of the discrete spectrum 
of A are nonzero real-analytic solutions of the equation 



O 

OS- AV' = AV' 



such that i^i'^z) = i/j{z) for all 7 in ro(A^) and such that 

\'il;{z)\'^dz < 00 



D 



;_i ■ where dz represents the Poincare measure of the upper half-plane. 



Let a be a cusp of ro(A^). Its stabilizer is denoted by Ta- An element a a G 
PSL{2,M.) exists such that aaC>o = a and a~^raO'a = Too- Let / be a ro(A^)- 
invariant function. If a is a cusp of ro(A^), then f{aaz) is Foo-invariant, and hence 
it admits a formal Fourier expansion 
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A ro(A'")-invariant function is said to be a Maass cusp form if it is square- integrable 
and is an eigenvector of A, such that the Fourier coefficient Cao(y) = for every 
cusp a of ro(A'"). If is a cusp form associated with a positive discrete eigenvalue 
A, then it has the Fourier expansion [5] 



where n = y^X — 1/4 and Ki^{y) is given by the formula §6.32, [18] 

^ ^ rV^ Jo (1 + ^2)-+^ 

I f°° f y, l,\ dt 



exp -^{t+ - 



The complex numbers pa{ni), 0) e Z, are called the Fourier coefficients of '0 
around the cusp o. 

The Hecke operators T„, n = 1, 2, • • • , (n, A?") = 1, which act in the space of 
automorphic functions with respect to ro(A), are defined by 



ad=n, 0<b<d 

An orthonormal system of eigenfunctions of A exists [5] such that each of them is an 
eigenfunction of all the Hecke operators. We call these eigenfunctions Maass-Heckc 
eigenfunctions. Let Aj, j = 1, 2, ■ ■ ■ , be an enumeration in increasing order of all 
positive discrete eigenvalues of A for ro(A) with an eigenvalue of multiplicity m 
appearing m times, and let Kj — — If ipj{z) is a Maass-Hecke eigenfunction 
of A associated with the j th eigenvalue A-,- , then 

{Tr^^^){z) = rj{n)^^{z) 

where Pjoo("^) = Pjooid)Tj{n) if m = dn with n ^ 1, (n, dN) = 1. 

Let Ex be a Hilbert space of functions spanned by the eigenfunctions of A with 
a positive eigenvalue A. The inner product of the space is given by 

(1.2) {F{z),G{z))= [ F{z)G{z)dz. 

JD 

The analogue for Maass forms of Eichler-Selberg's trace formula [11], p. 85 for mod- 
ular forms is obtained for the full modular group in [8] . In this paper, the trace trT„ 
of Hcckc operators acting on the space E\ is computed for congruence subgroups 
ro(A). Some of this computation is implicit in Hejhal [3]. 

Denote by hd the class number of indefinite rational quadratic forms with dis- 
criminant d. Define 

(1-3) ed = 2 

where the pair (vq, uq) is the fundamental solution [9] of Pell's equation v'^—du^ = 4. 
Denote by fi the set of all the positive integers d such that d = or 1 (mod 4) and 
such that d is not a square of an integer. 
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Theorem 1. Let N be a square free positive integer, and let n be a positive integer 
with (n,N) = 1. Put 

T (^\-\^ \^ j.i-2sM(!M0) \^ \^ M ^ Ined 

m\N k\N ^ ' ^ den u ^ ^ ^ ^ 

for > 1, where the summation on u is taken over all the positive integers u such 
that V4n + dk'^u^ G Z. Then Ln{s) is analytic for > 1 and can be extended by 
analytic continuation to the half-plane > except for a possible pole at s = 1/2 
and for possible simple poles at s = 1, | ± inj, j = 1, 2, • • • . For any eigenvalue 
A > of the Laplace- Beltrami operator for Tq[N), we have 



trTn = 2n"^ReSs^ii2+ii^Ln{s), 



where k = \/\ — 1/4. 



The paper is organized as follows. In section 2, we recall the theory of Selberg's 
trace formula [11]. Elements of the set F*, which is defined in section 2, can be 
divided into four types, the identity, hyperbolic, elliptic and parabolic elements. 
Next, in section 3 we compute contributions of the identity, elliptic, hyperbolic and 
parabolic elements to the trace formula. A technical part of this section (Lemma 
3.3 - Theorem 3.9) is to compute the contribution from hyperbolic elements whose 
fixed points are cusps of ro(A'"), and the result is given in Theorem 3.9. By using 
the Selberg trace formula and by considering the contributions to the trace formula 
of the identity, elliptic , parabolic elements and hyperbolic elements whose fixed 
points are cusps, we obtain the analyticity information about a series formed by 
contributions to the trace formula of hyperbolic elements whose fixed points are not 
cusps, and a precise statement is given in Theorem 3.12. In section 4, we compute 
explicitly the total contribution to the trace formula of hyperbolic elements whose 
fixed points are not cusps. The result is stated in Theorem 4.6. A technical part of 
this section is to relate the number of certain indefinite primitive quadratic forms, 
which are not equivalent under ro(A^), to the class number of indefinite primitive 
quadratic forms for the full modular group S'L2(Z), and the relation is given in 
Lemma 4.5. Finally, the main theorem of this paper follows from Theorem 3.12 and 
Theorem 4.6. 

The authors wish to thank Atle Selberg for his valuable suggestions during prepa- 
ration of the manuscript, and wish to thank William Duke for the reference [2]. 

2. The Selberg trace formula 
Let s be a complex number with 3fJs > 1. Define 

= + 
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and 

v'|2 



k{z, z') = ^■ 



z — z 

yy' 



for z = X + iy and z' = x' + iy' in the upper half-plane. Then k{mz, mz') = k{z, z') 
for every 2x2 matrix m of determinant one with real entries. The kernel k{z, z') 
is of (a)-(b) type in the sense of Selberg [11], p. 60. Let 



poo 

l{u)= k{t)- 
J w 



dt 

9[ 



-w 

with = + e-'^ - 2. Write 



J —oo 

Then 

(2.1) g{u) = V^j^{t+-) - -j== = c(l + -) 2 

where c = 2^/^r{s - |)r-i(s). Since 

POO -I 

h(r)^c4'-^ / {u + - + 2)^-'u''^-^du 
Jo u 

= c4'-^ /"°"(w + - + 2)^--(M^^ + «-^n — 
Ji u ' u 

4*(s - -) 



where A{r, s) is finite for \Qr\ ^ 1/2 and for 3?s > 0, we obtain that 

(2.2) hm {s--- iK)h{r) = { ^ ''^'/^+^^^ ' ' 

s^i/2+iK 2 1^ 0, for r 7^ ±k. 

Remark. Atle Selberg told the second author that he used the function k{t) = 
(1 + 1)"'' in some of his unpublished works, which was convenient for computations 
(cf. Selberg [13]). 

Let n be a positive integer with (n, A^) = 1. Define 

r = Uad=n^('l "Mro(iv). 

o<b<d^ \ a J 

Since (n, A^) = 1, we have that e F* whenever T e F*. Every element of F* is 
represented uniquely in the form 

-")-, 
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with ad = n, < b < d and 7 e ro(A^'). It follows that T* satisfies all the require- 
ments given in [11], p.69. Let Oi, a2, • • • , a^(N) with iy{N) = ^^\n,w>o ^ ((^' 5)) 
be a complete set of inequivalent cusps of ro(A'^), where (p is Eulcr's function. We 
choose an element G PSL{2,M.) such that ctu.cxd = and a'^^Va^Cai = ^00 for 
i = 1,2, ■ ■ ■ , z^(iV). The Eisenstein series Ei{z, s) for the cusp aj is defined by 

7er„.\ro(Ar) 

for > 1 when z is in the upper half-plane. Define 

K{z,z')^ J2 k{z,Tz') 
Ter* 

and 

H{z, z')=J2 E 4^ / h{r)E,{^^, - + ir)E,{z', - - ir)dr. 

i=l ad=n,0<b<d ~°° 

Denote by tvjTn the trace of the Hecke operator acting on the space E\ .. It 
follows from (2.14) of [11], the argument of [7], pp. 96-98, Theorem 5.3.3 of [7], and 
the spectral decomposition formula (5.3.12) of [7] that 




for 3fJs > 1, where d{n) is the sum of positive divisors of n. 



3. Evaluation of components of the trace 

For every element T of F*, denote by Fy the set of all the elements of Fo(A'') 
which commute with T. Put Dt = Tt\H. The elements of F* can be divided into 
four types, of which the first consists of the identity element, while the others are 
respectively the hyperbolic, the elliptic and the parabolic elements. If T is not a 
parabolic element, put 

c(T) = [ k{z,Tz)dz. 

3.1. The identity component. 

If F* contains the identity element /, then 

c(/)= I dz. 

JTo{N)\n 
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3.2. Elliptic components. 

There are only a finite number of elliptic conjugacy classes. 



Lemma 3.1. Let R be an elliptic element ofF*. Then 

kit) 



'^-^^ " 2m sin ^ io 



t + 4sin^^ 



:dt, 



where m is the order of a primitive element of Tji and where is defined by the 
formula trace{R) = 2 cos 6*. 

Proof. Since R is an elliptic element of F*, an element a e PSL(2,M.) exists such 
that 

„ _i / cos^ — sin^\ ~ 
aRa = \ = R 

\ sm B cos B J 

for some real number < < tt. Denote by {aTo{N)a~^)^ the set of all the 
elements of aTo{N)a~^ which commute with R. We have 



c{R) = / k{z,Rz)dz 
Jd~ 



where = {aTo{N)a-^)f^\n. 

Let 7 = ^"^^ be an element of ro(A'") which has the same fixed points as 

R = Then {a — S)c = 7(0 — d) and Pc = 76. It follows that 7 commutes 

with R. By Proposition 1.16 of [14], a primitive elliptic element 70 of ro(A'") exists 
such that {r]rr]~^)^ is generated by tj^qtj^^ . Since rjjor]~^ commutes with R, it is 
of the form 

cos ^0 — sin 6*0 
sin ^0 cos do 

for some real number 6*0. By Proposition 1.16 of [14], 9o — n /m ior some positive 
integer m. It follows from the argument of [7], p. 99 that 

By the argument of [7], p. 100 we have 

^ i 
2msme Jo y/t + Asin'^e 
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3.3. Hyperbolic components. 

Let P be a hyperbolic element of F*. Then an element p exists in S'L2(R) such 
that 

with Ap > 1. The number Ap is called the norm of P, and is denoted by NP. It 
follows that 

c(P) = I k{z, NPz)dz 
Jd~ 

where Dp = (pTo{N) p~^) p\H. Let Pq be a primitive hyperbolic element of S'L2(Z), 
which generates the group of all elements of 5'L2(Z) commutating with P. Then 
there exists a hyperbolic element Pi G ro(A^), which generates Fp, such that Pi is 
the smallest positive integer power of Pq among all the generators of Fp in Fo(A^). 
Detail discussion about the "primitive" hyperbolic element Pi is given in the proof 
of Theorem 4.6. 

Theorem 3.2. Let P be a hyperbolic element ofV* such that Fp ^ {12}- If Pi is 
a "primitive" hyperbolic element ofTo{N) which generates the group Tp, then 

In NP 

c(P) = 7 rn^ -^——7770 (In ATP). 

^ ^ (iVP)V2 _ (iVP)- 1/2^^ ^ 

Proof. An argument similar to that made for the elliptic elements shows that every 
element of Fo(A'"), which has the same fixed points as P, commutes with P. Because 
pPip~^ commutes with P, it is of the form 




for some real number Ap^ > 1. Then 

^ ' Ji y'J-^ \ NP 

The stated identity follows. □ 

We next consider c(P) for hyperbolic element P e F* with Fp = {12}- 

Lemma 3.3. Let P = (c d) ^ hyperbolic element ofT* such that Fp = 
{I2}. Then fixed points of P are cusps ofTo{N). Moreover, Fp = {I2} if and only 
if A + D = I ("^ + 7^) for some divisor m of An with m 7^ for C ^ 0, and if 
and only if A ^ D for C = 0. 

Proof. Let P a hyperbolic element of F* such that Fp = {12}- If the fixed points 
of P are not rational numbers or infinity, then they are zeros of an irreducible 
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polynomial ax^ -\-hx-\-c with a = mod(A^). \i d=h^ — 4ac and {u, v) is a solution 
of Pell's equation — dv? = 4, then 



^ -cu 
au ^ 



belongs to ro(A'") and has the same fixed points as P, and hence it commutes with 
P. This contradicts toFp = {12}- Therefore, fixed points of P are rational numbers 
or infinity. \iu/w is a fixed point of P, then 



/ 1 + Nuw -Nu^ \ 
\ Nw'^ 1 - Nuw ) 



is a parabolic element of ro(A'") and has u/w as a fixed point. Hence, fixed points 
of P are cusps of ro(A^). 

Conversely, if T is an element of F* having two distinct fixed points with at least 
one of them being a rational number, then Tt = {^2}- Otherwise, let ri,r2 be 
the fixed points of T, and let 7 e To{N) (7 I2) with 7T = T7. Then either 
7 or 7^ (7^ I2), say 7^, has ri,r2 as its fixed points, and hence it is a hyperbolic 
element of ro(A^). Since rational points are cusps of ro(A^), by Proposition 1.17 of 
[14] 7^ is parabolic. This is a contradiction, and hence we have Ft = {12}- Thus, 

we have proved that Ft = {12} for an element T = ^ j G F* if and only if 

a + d = ^(to + ^) for some divisor m of 4n with m ^ ii c ^ Q or a ^ d ii 
c = 0. □ 

Every cusp of ro(A'") is equivalent to one of the following inequivalent cusps 

Vi 

(3.1) — with u, w > 0, (w, w) — 1, wlN. 

w 

Two such cusps u/w and ui/wi are Fo(A^)-equivalent if and only if w = wi and 
u = ui modulo {w,N/w). Let a = u/w be given as in (3.1). By (2.2) and (2.3) of 
Deshouillers and Iwaniec [1] , we have 

(3.2) r. = |(^ / c^O(mod[«;^iV])| 



and 



where 



(TaOO = and ""^PaCa = T 



(3.3) 



_ (a^ JW\N ] \ 
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Let P be a hyperbolic element of F* such that Tp = {12}- Assume that a is a 
fixed point of P. Then 00 is a fixed point of a~^P(Ta, and hence there exist positive 
numbers a, d with ad = n,a ^ d such that P is of the form 

^ "VO ' V^\{a-d)^-b[w^N] d + b[w^N]^J- 

We claim that a, d are integers. Since P = (^c er*,we have A, B,C,D e Z, 
AD-BC = n, and C = mod(iV). By (3.4), we find that 



(3.5) a = D + C-, d=D-B-, b = B „. „ 
and 

(3.6) A-D = C--B-. 

w u 



Since {w,u) = 1, we have u\B by (3.6). It follows from (3.5) that a,d are integers. 
Since c(P) depends only on the conjugacy class {P} represented by P, we can 
replace P by ^~^P^ without changing the value of c(P). Replacing P by j~^P^ for 
some element 7 e F^, we can assume without loss of generality that ^ 6 < |a — 
in (3.4). 

Lemma 3.4. Let a = u/w be given as in (3.1). Then 

p I fa b\ _i 

with a, d & 'Z, ad = n, a d is a hyperbolic element ofT* withTp = {I2} and P(a) = 
a if and only if (w, N/w)\{a — d) with b being chosen so that (a — d)^ — b[w'^, N] is 
divisible by N. 

Proof. Let P be a hyperbolic element of F* with Fp = {I2} and P(a) = 0. By 

(3.4), [a — d)^ — b[w'^, N] is divisible by N, and there exists an integer k such that 
■^N\a — d — kw. This implies that {w, N/w)\a — d, which can also be seen directly 
from the identities P(a) = a and a + d = ^(m + ^) for some divisor m of An. 

Conversely, let a,d E Z'^,ad — n,a ^ d and {w,N/w)\a — d. Write a — d = 
£{w,N/w) for some integer £. Since uN/w{w, N/w) and w/{w,N/w) are coprime, 
there exist integers A and r such that XaN + rw = {w, N/w), and hence £XaN = 
a — d — irw. This means that there exists an integer k such that —N\a — d — kw. 
If we choose A = a — kw, B = ku, C = ^{a — d — kw) and D = d + kw, then 

AD — BC = n and C = mod(A^). Let P = {^cd\ Then P can be expressed 
as in (3.4), P(a) = a, and Fp = {I2}. Let 7 = C/(C, D) and 6 = D/{C, D). There 
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exist integers a, (5 such that a5 — (3'^ = 1. Since (n, A?") = 1, we have 7 = mod(A'"). 
It follows that 

J_/^ Ba-A(3\(a 
^\ {C,D) ) 5. 

This implies that P e F*. Thus, we have proved that 

J, I (a h\ _i 



with a, d & Tj, ad = n, a ^ d is an element of F* with Fp = {I2} and P{u/w) = u/w 
if and only if there exists an integer k such that —N\a — d — kw, that is, if and only 
if (w, N/w)\{a — d) with h being chosen so that {a — d)^ — h[w'^, N] is divisible by 
N. □ 

Lemma 3.5. Let <x = u/w be given as in (3.1). Let a,d & 'Z'^,ad = n, a ^ d. If 
{w, N/w)\a—d, then there are exactly \a—d\ number ofTQ{N)-inequivalent hyperbolic 
elements P e F* with P(a) = a and Fp = {I2}, which are of the form 

1 / a 6 \ _i 



P = -^aa . , (T, 



a 



with ^ b < \a — d\. 

Proof. For a, d & 'Z, ad = n, a ^ d, let 



1 fa b\ _i , p, 1 fa b'\ _i 



be two elements of F* with ^ b,b' < \a — d\ such that Fp = {I2}, P{o) = and 
Fp/ = {l2},P'(a) = o. We claim that b' — b + £ for some integer i. In fact, for 
P there exists an integer k such that —N\a — d — kw, and for P' there exists an 
integer k' such that — d — k'w. Then we have ^N\{k' — k)w, that is, 

k' = k + 



w{w, N/w) 

for some integer i. By using (3.4), we find that 

[t(;2,iV] (w,N/w) 

The claim then follows. Conversely, given an integer £ with 0^6 + £<|a — (i|,let 
6' = 6 + £ and 

- , . £uN 
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Put A' = a- k'w, B' = k'u, C'=^{a-d- k'w) and D' = d + k'w. Then 

is a hyperbolic element of F*, Vpi — {I2}, and -P'(a) = a. Moreover, if 6' 7^ 6 
modulo \a — then P' and P are not ro(A^) -equivalent. Otherwise, an element 

7 e ro(iV) exists such that P'7 = 7P. Put 7 = ( J *) and a-^7aa = (^5)- By 

using the relation P'7 = 7P, we obtain that {a — d)(3 = ah — Sb' and ry = 0. Since 

7 e ro(iV), by using 77 = and the relation a~^'yaa = (05) that a and 5 

are integers, and hence a = S = 1. Furthermore, we obtain that /3 = -C/[w'^, N]. 

Since (j~^7(Jci = Q^j? we have (cr~^7(ja)(oo) = 00, that is, 7(a) = o. Hence 

7 e Fq, which is given by (3.2). This implies that P is an integer. It follows that 
{a — d)\{b — h'). This is a contradiction. Therefore, for fixed a, d e 1^'^ ^ad = n^a ^ 
d, there are exactly |a — d| Fo(A'^)-inequivalent hyperbolic elements P in F* with 
P{u/w) = u/w and Fp = {I2}) which are of the form 

with0^6< |a-(i|. □ 

Lemma 3.6. Let a = u/w, and let 

&e tw;o hyperbolic elements ofT* with P{<x) = a, P'(o) = 0, Fp = {I2} and Fp/ = 
{I2}. Then P is Vq{N)- conjugate to P' for some number b' if and only if the two 
fixed points of P are Vq{N)- conjugate. That is, P is Tq{N) -conjugate to P' for 
some number b' if and only if 

fid-a)^ + b[w'^,N] \ 

where i is the greatest common divisor of ^[to^, N] ^ and — a) ^ + ^[to^. A'"] . 

Proof. We first assume that the two fixed points of P are Fo(A'")-conjugate. That 

is, there exists an element T = " f j ^ ro(A'"), which is not the identity, such that 

a,T(a) are the two fixed points of P. Let P' = T'^PT. Then P'(a) = a. Since 
Fp = {I2}, we have Fp/ = {12}- We claim that P' is of the form given in the 
statement of the lemma. 
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We can write P' in the form 



P' = ^(Ja 



1 fa' b'\ _i 



with a' , d! e Z"*" and a'd' = n. Replacing T by T7 for some element 7 e r^, we 
assume that ^ b' < \a' - d'\. Since P' = T~^PT, we have 

In general, from the identity 

A B\ fa' b'\ ^ fa b\ f A B 
C d)\Q d' ) ~ \Q d) \C D 

with AD-BC 0, we obtain that {d-o!)C = 0, {d-d')D = b'C and {a' -a) A = bC. 
If C = 0, we have a' = a and d' = d, and hence P' is of the form 



("' 








I 


d' 


= (0 





Since P' is ro(A^)-conjugate to P and ^ 6' < |a — d\, we must have P' = P hj 
the proof of Lemma 3.5. That is, T e Fp, and hence T — I2. A contradiction is 
then derived. Therefore, we must have C 7^ 0. It follows that a' = d. Since ad = n 
and a'd' = n, we have d' = a, and our claim follows. 

Conversely, suppose that there exists an element T = (^q^^ ^ ro(A'") such that 

T-^PT = P', that is. 



If we write 



D-B^ 



then (3.7) is equivalent to the system of equations 

({a-d){A + B^) = b[w^ N]^iA + B^-C^-D), 
I b{D-B^) = b'iA + B^). 

Write this system of equations as 

r {{a- d)^ -b[w\N])uiA + B^) = b[w\N]^{-Cu- Dw), 
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The first equation of (3.8) can be written as 





T(a) = 



1 - 



a—d 



b[w'^,N]a 

which is the second fixed point of P. That is, the two fixed points of P are ro(A^')- 
conjugate. Since ci = ^ and 

b[w^, N]u/w 
{d-a)^ + b[w'',N] 

are the two fixed points of P, the last statement of the lemma follows from Lemma 
3.6 of Deshouillers and Iwaniec [1]. □ 

Lemma 3.7. Let a,d & Z+ja ^ d,ad = n, and let a = u/w be given as in (3.1) 

with {iv,N/w)\{a — d). Assume that a and a! — ui/wi are the two distinct fixed 
point of a hyperbolic element P G F*. Then {w' , N/w')\{a — d) forw' = {wi,N). In 

other words, if P = "^cto (o d) '^a^ ^ hyperbolic element ofV* with P(a') = a' , 

then it can also be written in the form 

1 (d b'\ _i 







a' 



Jn 

for some number b' . 

Proof. It follows from the argument made in the paragraph preceding Lemma 3.4 
that there exist positive integers a', d' with a'd' — n such that 

for some number b' . Then wc have 

(3.9) ^a'^»'(o d') = {o d)''^'''^'- 

Write (J-Va' = [cd)- fo^ows from (3.9) that {d - a')C = 0, (a' - a)A = bC 
and (d — d')D — b'C. Since a ^ a', by using (3.3) we find that C 7^ 0, and hence we 
have a' = d. It follows that d' = a, that is, P can also be written in the form stated 
in the lemma. Then by the argument made in the first paragraph of the proof of 
Lemma 3.4, we have {w', N/w')\{a — d). □ 

For a large positive number Y , define 

Dy = {zeD : ^(J-^z < y, z = 1, 2, • • • , /i}. 

Let 

{Dp)y = U iDy. 
7ero(iv) 

Write 

c{P)y = / k{z, Pz)dz. 
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Lemma 3.8. Let a = u/w be given as in (3.1), and let P = ■^o'a (o d) '^a^ with 

a,d E Z+, ad — n, a ^ d be a hyperbolic element ofF* withVp — {I2} and -P(a) = a. 
Then 

\a-d\ d^ \ n J ^FT^ ^ ' 

where o{\) ^ a.s y ^ 00 and p = jiyC^ j e ,N\Y with C = {a - d)^ -b[w^, N] 
and with £ being given as in Lemma 3. 6. 

Proof. Let 

^ p r 



7 



q s 



be an element of 5'L2(M). The linear fractional transformation, which takes every 
complex z in the upper half- plane into 7(2;), maps the horizontal line = Y into 
a circle of radius with center at | + ^^ly • Let 



a—d 

1 



Then 



Note that 



pi 0'\ -1 



V^^'^^ \o d 



w/u _ b^/[w^,N] buy/[w2,N] 
y^[w^,N] O'-'^ {a-d)w 



Since 



p_J_^ 1/1/2/ OWa Q\ y 

for any y ^ 0, by choosing y — {a — d) ^/\w^~TT\ we obtain that 

k{z, —z)dz 

where 



{a-d)^-b[w\N] h[w\N]l 

f^a — I -1 u 



a — d {a — d)% 

The linear transformation z aaZ maps the half-plane '<sz > Y into a disk Da 
of radius 2[w^^n]y ^^^^ center at a-|- 2[w2V]y Then the transformation z — >• //a(^) 
maps the disk Da into the half-plane '^z > {a — d)^[w'^, N]Y. Let 

^ _ —b[w'^, N]u/w 



{a-d)^-b[w\Ny 
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If 



then CTpOO = p and cr^^Fpcrp = Too- By the definition of Dy, the image of the 
half- plane '<sz > Y under the linear transformation z ^ apZ is not contained in 
{Dp)y- Since the linear transformation z {fia<yp){z) maps the half-plane '^z > Y 
into a disk Dp of radius p centered at ip, where p = C^/2[C^/£^, N]Y, the disk Dp 
is not contained in /Xa{(-Dp)r}. It follows that 

/.n p{a-df[w^,N]Y/sin9 /<'„_J\2\ J„ 

ciP)Y= / de k ^ + oil) 

Jo J2psine \nsin e J rsm 6 



i: 



k ( ^J^t) M(a-rf)-[^iV]Yt/2,}^^ 

* n J ^ ' 



'1 

where o(l) has a limit zero when y — > oo, and hence we have 

v/i?ln{(a- rf)^K,Af]y/2p} g f , f ili^A JSL*+o(l). 

|a - a| a Ji \ n ) \Jt — \ 

This completes the proof of the lemma. □ 

In the rest of the paper, we shall indicate explicitly when we assume that N is 
square free. 

Theorem 3.9. Let N be a square free positive integer. Then we have 

{P},rp = {l2} ad=n,d>0,a^d 

+ — E E ~ ^^'"'^t^' ^^^^'^ ff(ln ^) 

tij|A/",ti;>0 ad=n,d>0,ay^d b 
ad=n,d>0,a^d ^ ' 

with C = (a — d)w — bwN , where £ = (C, bN) and is over the \a — d\ numbers 
such that C, bN/w e N\C and ^ b < \a-d\. 

Proof. Let P and P' be hyperbolic elements in F* such that Fp = {I2} and Fp/ = 
{I2}. Assume that P' and P are Fo(A^)-conjugate, that is, T~^P'T = P for some 
element T e Fo(A'"). If Oi,a2 are the two fixed points of P, then T(ai),T(a2) are 
the two fixed points of P'. In other words, if at least one of the two fixed points 
of P is not Fo(A^) -conjugate to a fixed point of P', then {P'} and {P} represent 
different Fo(A'")-conjugacy classes. 
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The stated identity then follows from Lemma 3.8. □ 

3.4. Parabolic components. 

Let /S be a parabolic element of F* . An argument similar to that made for the 
elliptic elements shows that every element of ro(Ar) which has the same fixed point 

as S commutes with 5". 11 a = u/w is the fixed point of S, then Vs = Fq, and hence 
we have a~^Vs(Ja = Too, where aa is given as in (3.3). Since (7~^S(7a commutes 
with every element of Foo , we have 



for some real numbers a, h with — n. An argument similar to that made in 
the paragraph following (3.4) shows that a is an integer. This implies that F* has 
parabolic elements only if n is the square of an integer. Furthermore, by (3.4) we 
see that elements of the form 



constitute a complete set of representatives for the conjugacy classes of parabolic 
elements of F* having a as its fixed point. It follows that 



where the summation on {S} is taken over all parabolic classes represented by 
parabolic elements whose fixed point is a = u/w and where o(l) tends to zero as 
y — > oo. Define 5n to be one if n is the square of an integer and to be zero otherwise. 

Theorem 3.10. Put 






c{oo)y = SnV{N) 




H{z, z)dz. 



Then 



c(oo)y 




ad=n,d>0,a^d 




J — oo 



/ h{r)-{l + ir)dr + -h{0){5nu{N) + d{n) ^ M^)} 



+ 




ir)dr + o(l) 
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where o(l) tends to zero as Y ^ oo and (fii{l/2) ^ oo (of. Selberg [13>]) 
Proof. By the argument of [7], pp. 102-106 we have 



= g{0) In(VnF) - — / h{r) — {l + ir)dr - g{0) ln2 + -h{0) + o(l). 
27r i 4 

Let 

Vij,m{s) = J2u^[Yl <^d/c) 

c ' ' \ d / 

where summations are taken over c > 0,d modulo c with (* ^) € cr~.^ro(A'")cra^. 
Then we have 



l-s 



where 



v^r(. - i) 

^y(s) = ^ij,o[s). 



By (1.1) and the Maass-Selberg relation (Theorem 2.3.1 of [7]), we obtain that 

f ^ ,az + b 

/ Ei{ — - — ,s)Ei[z,s)dza 

JDy ^ 

^ "^ y,j(a/d)^y^+^-- ^ - ipij{s)ifij{s){a/dY-'Y^-'--' 



s + s - 1 

^ (a/d)V.,(a)y---- - ^,j{s){a/dY-^Y-^-\ ^ 

s — s 

for nonreal s with 3fJs > 1/2, where o(l) tends to zero as F ^ oo. Hence we have 
f ( Ei{^^^,s)\Ei{z,s)dz 

^ - Ejiy <Pij{s)<PiMY^-^--^ ^u{^Y^-' - ^u{s)Y-^-\ 

^ s+s-l s-s ^ 

ad=n,d>0 
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By partial integration, we obtain 



h{r) = 4 / 9^^\^^u)u'''-^du 
f Jo 



for nonzero r. Then it follows that 
(3.10) 



lim / h{r){ [ I y Ei{^^^,S + ir)] Ei{z,S -ir)dz}dr 
= V^ {47r5'(ln^)lny- 5^ f %) (^)" ^-. (^ + ^r)(^y - zr)dr 

ad=n,d>0 J = l 

+ J_Jir)[-^) dr} + oil). 

By the Riemann-Lebesgue theorem (cf. §1.8 of [16]), we have 

(3.11) 



oo 

2 



f A A sinirln aFV^^ } , 



By (3.10) and (3.11), we have 



H{z,z)dz=Y, Yl {^(ln^)lny+^/.(0)^i,(^) 



The stated identity then follows. □ 

Let A'^ be a square free positive integer. It follows from Theorem 3.9 and Theorem 
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3.10 that 



lim 

Y^oo 



( 

c(oo)y + E 
V {P},Vp={l^} 



u{N) 



1=1 

-,2 



+ 1" E E E 

{6AZ) w\N,w>Oad=n,d>0,a^d b ' ' 

h[r) — [l + %r)ar 



27r ./_^ ' ' r 



+ E E ^r''wQ'V;,(i+-M.(i-f)<*- 

i,i = l ad=n,d>0 •'-oo 

with C = (a — — bwN, where £ = (C, 6A^) and the summation on b is taken over 
aU numbers b such that C, bN/w e Z,N\C and ^ 6 < |a — Note that there are 
exactly |a — ci| number of such numbers b by Lemma 3.5. 

Denote by c(oo) the right side of the identity (3.12). We conclude that the trace 
formula (2.3) can be written as 

oo 

(3.13) 

= c(/) + Ec(i?)+ E c(P) + c(oo) 

{R} {P},Tp^{U} 

for 3?s > 1, where the summations on the right side of the identity are taken over 
the conjugacy classes. 

Lemma 3.11 (Hejhal [3]). (cf. Proposition 13.6 of Iwaniec [Q]) Let N be a 
square free positive integer. Then, for any pair of cusps = l/wi,aj = 1/wj with 
N = ViWi = VjWj, we have 

(Pij{s) = (f{s)pij{s) 

where 
and 

p,,is) =^iiv,,v,)iw„w,))l[{p^^ - l)-i n iP'-p'~n- 

p\N p\{wi,Vj){wj,Vi) 
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Theorem 3.12. Let N be square free, and put c{P) = /p^^-^ k{z, Pz)dz for hyper- 
bolic elements P e F*. Then the series 

{P},rp/{i2} 

represents an analytic function in the half-plane > except for a possible pole 
at s = 1/2 and for possible simple poles at s = 1, | ± inj, j = 1,2, ■ ■ ■ . 

Proof. We have 

where A{s) is an analytic function of s for 3fJs > and where Os{u~^) means 
that, for every complex number s with > 0, there exists a finite constant B{s) 
depending only on s such that 

\Os{u-^)\ ^ B{s)u-^. 

Moreover, for every fixed value of u, the term Oa{u~^) also represents an analytic 
function of s for 3?s > 0. Since 

h{r) = — / g^^\\nuy''-Uu 
f Jo 

for nonzero r, we have 

\s — — ir s — -^^%r ) 

for 3?s > 1 and for nonzero r with \^r\ < | — e. By analytic continuation, we obtain 
that 

(3-14) h{r)= ^f'%}\, +Os{r-^) 

for 3fJs > and for nonzero r with |$5r| < ^ — e. It follows from results of [17] that 
the left side of (3.13) is an analytic function of s for > except for simple poles 
at s = 1, 1 ± iKj, j — 1, 2, • • • . Then the right side of (3.13) can be interpreted as 
an analytic function of s in the same region by analytic continuation. 

Since k{t) = (1 +t/4)~*, by Lemma 3.1 we have that c{R) is analytic for 3?s > 
except for a simple pole at s = 1/2. There are only a finite number of elliptic 
conjugacy classes {R}. The term c(/) is a constant. 
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Since g{0) = 2y^r{s - ^)r{s)-^, 



T{s) Ji u 



s 

U 



and 

s(ln5) = 2V?4-.^(- + - + 2j , 

the sum of first three terms on the right side of the identity (3.12) is analytic for 

> except for a pole at s = 1/2. 

Since k{t) = (1 + 1/4)~^, the sixth term on the right side of the identity (3.12) 
is analytic for 3?s > except for a pole at s = 1/2. 
By Stirling's formula the identity 

(3.15) ^=ln^ + 0(l) 

holds uniformly when | argz] < tt — 5 for a small positive number S. It follows from 
(3.14) and (3.15) that the fourth term on the right side of the identity (3.12) is 
analytic for 3fJs > except for a possible pole at s = 1/2. 

By the functional identity of the Riemann zeta function C{s), we have |<^(s)| = 1 
for JJs = 1/2. This implies that 



(3.16) <Pij{s)<P^A^) = ^\P^As)f +P'^,{S)P^,{S) 



for 3fJs = 1/2 by Lemma 3.11. The identity 

^ ^ ^ ^(«) ~ r(.) r(i-.) 'c(2.) 'c(2-2.) 

holds for 3fJs = 1/2. It follows from (3.14)-(3.17) and the formula for Pij{s) given in 
Lemma 3.11 that the fifth term on the right side of the identity (3.12) is analytic 
for 3fJs > except for a possible pole at s = 1/2. 

Therefore, by (3.13) we have proved that the series 

{P},rp^{i2} 

represents an analytic function of s in the half-plane > except for a possible 
pole at s = 1/2 and for possible simple poles at s = 1, | ± inj, j — 1,2, ■ ■ ■ . □ 
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4. Proof of the Main Theorem 

Lemma 4.1. Let X > be an eigenvalue of A for Tq^N) with N square free. 
Assume that (n, N) = 1. Put r = 1/2 + in with k = — 1/4. Then we have 

VV^^^trTn = lim(6' - r) V c(P) 
^ ^ {P},rp^{U} 

where the right side is defined as in Theorem 3.12. 
Proof. By (2.2), (3.14) and results of [17], we have 

oo 

lim(s - r) V h{Kj)tTjTn = 0. 
By the proof of Theorem 3.12, we have 



lim(s-T) d{n)h{--) - c{I) - c{oo) -J2c{R) = 0. 
\ ^ {R} J 

Then the stated identity follows from (2.2), (3.13) and Theorem 3.12. □ 

A quadratic form ax^ + bxy + cy^, which is denoted by [a, b, c], is said to be 
primitive if (a, 6, c) = 1 and 6^ — 4oc — d E ^. Two quadratic forms [a, b, c] and 
[a' , b', c'] are equivalent if an element 7 e 5'L2(Z) exists such that 

a' b'/2\_ ,( a b/2\ 
b'/2 c' J ~^ \b/2 c J^' 

where 7* is the transpose of 7. This relation partitions the quadratic forms into 
equivalence classes, and two such forms from the same class have the same discrim- 
inant. The number of classes hd of primitive indefinite quadratic forms of a given 
discriminant d is finite, and is called the class number of indefinite quadratic forms. 

Remark. Siegel [15] proved that 

(4.1) lim ^ 1. 

d^oo Ind 2 

Lemma 4.2. P is a hyperbolic element ofT* with Tp ^ {I2} if and only if there 
exists a primitive indefinite quadratic form [a, b, c] of discriminant d such that 
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with 

- {dN^{a, N)-'^}u'^ = 4n. 
If Xp is an eigenvalue of P, then 

. 1 ^ Nu Vd 

Xp — - — = i 



Let 



Xp (a, N) ^ 



where the pair (vq, uq) is the fundamental solution of Pell's equation — du^ = 4. 
Then P is Vq{N)- conjugate to a hyperbolic element P' e F* with Tp> ^ {I2} if and 
only if Pq is To{N)- conjugate to Pq, where Pq is associated with P' similarly as Pq 
is associated with P. 

Proof. Let 

be a hyperbolic clement of F* such that Fp 7^ {12}- Then fixed points ri, r2 of P are 
not rational numbers by Lemma 3.3, which satisfy the equation Cr^ + {D—A)r—B = 
0. This implies that Fp is the subgroup of elements in To{N) having ri, r2 as fixed 
points. Let a = C/n,b = {D — A)/fx and c = —Bf/j,, where /i = [C^D — A,—B). 
Then [a, 6, c] is a primitive quadratic form with ri, r2 being the roots of the equation 
ar^ + 6r + c = 0. By Sarnak [9], the subgroup of elements in SL2{'L) having ri, r2 
as fixed points consists of matrices of the form 

^ -cu 
an ^ 

with — du^ = 4 where d = b^ — 4:ac, and it is generated by the primitive hyperbolic 
element 

-cuo 



where the pair {vq, uq) is the fundamental solution of Pell's equation — du^ = 4. 

Since P and Pq have the same fixed points, we have A = D — bC/a and B = 
—cC/a. Since P belongs to F* and AD — BC = n, C satisfies 



(4.2) 



aD^ - bDC + cC^ = na 
a\C, N\C. 

Let Xp be an eigenvalue of P. Then it is a solution of the equation A^ — "^J^ A+1 = 0. 
By using A — D — bC/a and B — —cC/a, we obtain that 

(4.3) ^-r = ±^ 

Ap ayn 
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and 

(4.4) Xp + ^ = ^{2D--C). 

Conversely, let a pair {C,D) be a solution of the equation (4.2). Define A 
D — bC/a and B = —cC/a. Then the matrix 

y/E\C D 

has the same fixed points as Pq, and eigenvalues of P satisfies (4.3) and (4.4). We 
have the decomposition 



v^V (L»,C) ; \C/iD,C) D/{D,C) 

where a and (3 are integers such that aD — f3C = {D,C) and where T = Q ^ ^ G 
ro(A'") is chosen so that 

n/{D,C) * \ fn/{D,C) -s 
{D.C))-' \ (AC) 

with < s < n/ {D, C). The first equation of (4.2) can be written as 

D''-hD- + cC- = n. 
a a 

Since {n,N) = 1 and a|C, we have {N, {D,C)) = 1. Hence (^^/(j) d/(^c)^ 
an element of ro(A''). Therefore, P is an hyperbolic element of F* with Tp {I2} 
by Lemma 3.3. 

Next, let V = 2D— 6^ and u = ^. Then the equation (4.2) becomes v'^—dv? = An 
with N\au. Since N\au, this equation can be written as 

(4.5) ^ -^ivF- 
Moreover, we have 

S\ /^^^^^^^^^^f^ -cNu{a,N)-' 

V^[C d) V^[aNu{a,N)-' v+muia,N)-^ 

Let Pq be the primitive hyperbolic element of /S'L2(Z) corresponding to [a' , b', c']. 
Since the identity 

holds for every element 7 G ro(A^), two forms [a, 6, c] and [a',6',c'] of the same 
discriminant are equivalent in ro(A^) if and only if an element 7 G ro(A^) exists 
such that 7~^Po7 = Pq- □ 
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Lemma 4.3. Let [a, b, c] be a primitive indefinite quadratic form of discriminant 
d, and let 

V ^Nu{a, N)-' v+bNuia,Nr' 

with — {dN^{a, N)~'^}u'^ = An be a hyperbolic element ofV* withTp ^ {12}- Let 
{vi^ui) with vi,ui > be the fundamental solution of Pell's equation v"^ — diu^ = 4, 
where di = dN"^ / (a, N)^ . Then Tp is generated by the hyperbolic element 



Pi 




C(a,7V) 



ofToiN). 

Proof. Let ri and r2 be the fixed points of P. Ttien tfiey satisfy ar'^ + br + c = 0, and 
Fp is the subgroup of elements in ro(A^) having ri, r2 as fixed points. By Sarnak 
[9], the subgroup of elements in 5'L2(Z) having ri, r2 as fixed points consists of 
matrices of the form 



^ -cu 

au ^ 



with v"^ — du^ = 4 where d = b^—4ac, and it is generated by the primitive hyperbolic 
element 



^0 - \^ 

where the pair {vq, uq) is the fundamental solution of Pell's equation — du^ = 4. 

Since Fp is cyclic, a solution vi,u[ > of Pell's equation v'^ — du'^ = 4 exists 
such that Fp is generated by 



vi—bu'^ 



Pl = 



2 —Cu'i 



and such that Pi is the smallest positive integer power of Pq among all powers of 
Pq belonging to Fo(A'"). Note that the eigenvalues of Pi are 



±\/du'^ 



Since Fp is generated by Pi, {vi^u'i) is the minimal solution of the equation — 
du^ = 4 with N\ au'i in the sense that (?;i + ^/dui)/2 is of the smallest value among 
all such solutions. Since A'"|a«^, we have j^fj^lu'i- Write 

Nui 
11, = 



26 



J. B. CONREY AND XIAN-JIN LI 



Then the pair (fi,wi) with vi,ui > must be the fundamental solution of the 
Pell equation v'^ - diu^ = 4, where di = dN^{a, N)~^. The stated result then 
follows. □ 

Two quadratic forms [a, b, c] and [a', b', c'] are equivalent in ro(Ar) if an element 
7 e To{N) exists such that 

a' h'/2\ _ t f a 6/2 



b'/2 c' ) ^ \h/2 c 

This relation partitions the quadratic forms into equivalence classes, and two such 
forms from the same class have the same discriminant. The number of such classes 
of a given discriminant d is finite, and is denoted by Hd- 

Lemma 4.4. Let [oj, 6j, Cj], j = 1, 2, • • • , H^, be a set of representatives for classes 
of primitive indefinite quadratic forms of discriminant d, which are not equivalent 
under Tq{N). Then we have 

{p}: Per*,rp^{i2} 

for > 1, where di = dN'^{aj, N)~'^ and where the summation on u is taken over 
all the positive integers u such that 4:n + dN'^{aj, N)~'^u^ is the square of an integer. 

Proof. It follows from (2.1) and Theorem 3.2 that 

^ - ^^^>r ^ Ap - 1/Ap y + — 4 — 

{p}: P6r*,rp/{i2} ^ ^ {P} I P \ 

for > 1, where Ap > 1 is an eigenvalue of P and Pi is given in Lemma 4.3. Let 
P be associated with a primitive indefinite quadratic form [a^, 6j, Cj] as in Lemma 
4.2. Then by Lemma 4.2, we have 

^ _ 1 _ Nu Vd 
^ Xp {aj,N)^/n' 

By Lemma 4.3, we have 

If P' is a hyperbolic of F* with Fp/ ^ {^2}^ and is associated with a primitive 
indefinite quadratic form [aj/ ,bj/ , cj/] as in Lemma 4.2, then P and P' are Fo(A^)- 
conjugate if and only if [aj, bj, Cj] and [a^/, bj/, Cji] are Fo(A^)-conjugate by the last 
statement of Lemma 4.2. Next, let T be a hyperbolic of F* with Ft 7^ {12}- Assume 
that T is associated with a primitive indefinite quadratic form [a, 6, c] as in Lemma 
4.2. If the discriminant of [a, 6, c] is not equal to the discriminant of [a^, bj, Cj\ which 
is associated with P, then P and T are not Fo(A'')-conjugate. The stated identity 
then follows. □ 
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Lemma 4.5. Let N be square free, and let k be a divisor of N. Then the number 
of indefinite primitive quadratic forms [a, b, c] with (a, N) = k of discriminant d, 
which are not equivalent under Vq{N), is equal to 



where d\ = dN^/P. 

Proof. Let [a, b, c] and [a', b', c'] be two indefinite primitive quadratic forms of dis- 
criminant d with (a, N) = k = (a', N). If they are equivalent under ro(A'"), then an 



This implies that b' = b (mod 2k). In particular, if 6' ^ 6 (mod 2/c), then [a, 6, c] 
and [a', 6', c'] are not ro(A^)-equivalent. 

Assume that q is an integer with 1 ^ ^ ^ 2A;. Denote by h-k,d,Q the set of 
representatives of indefinite primitive quadratic forms [a, 6, c] of discriminant d with 
{a,N) = k and b = g (mod 2k), which are not equivalent under ro(A^). Let 
£^ gN/k °^ representatives of indefinite quadratic forms [aN, 6, c] of 

discriminant di with (a, 6, c) = 1, {N, b, c) = N/k and b = gN/k (mod 2A'"), which 
are not equivalent under ro(A^). 

A map from Ak,d,g to C%^^^^^j^^f^ is defined by T : [a, b, c] [aN/k, bN/k, cN/k]. 
We claim that T is bijective. By the definition of Ak,d,gi we see that T is injective. 
Conversely, if [aiN,bi,ci] is an element of jC^ gN/ki then we have (ai,6i,ci) = 
1, (TV, 6i, ci) = N/k, bi = gN/k (mod 2N) and 6? - ANa^ci = di. Let [oq, bo, cq] be 
an element of h.k,d,Q- Then 




element 




^ = «(a/?+ ^5) + 7(^/5 + c5). 



bl = (boN/k)^ + AlboN— + AfN"^ 



for some integer l. Since 



{boN/k)'' - A^N ■ CO 



N 



we have 



{boN/k)^-A'^N-co 



N 



di = (boN/k)^ + MboN— + M'^N^ - ANaici. 



That is, the identity 



(4.7) 
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holds for some integer l. Note that {qq^N) = k. Since {N,bi,ci) = N/k and 
(ai, 61, ci) = 1, we have {ai,N/k) = 1, and hence it follows from (4.7) that N/k\ci. 
Let a = ciik, b = bik/N and c = cik/N. Then we have (a, A^) = k,b = g (mod 2k) 
and d = b^—4ac. We claim that (a, 6, c) = 1, that is, (ai/c, bik/N, cik/N) = 1. Since 
(ai,6i,ci) — 1, it is enough to show that (A;,6i,ci) = 1. Since (A*", 6i,ci) = N/k, 
we must have (/c,5i,ci) = 1, and therefore we have (a, 6, c) = 1. Thus, [a, 6, c] is 
an element of Ak,d,Q, and T maps it into the element [aiN,bi, ci] of gN/k- 
Therefore, T is surjective. Thus, we have proved that T is a bijection if the set 
Ak.d,e is not empty. By Proposition, p. 505 of Gross, Kohncn and Zagier [2], the 
number of elements contained in ^^jj^ is hd^, and hence the set A.k,d,Q contains 

hd^ elements if it is not empty. 

If Qi ^ Q2 (mod2fc), then the set of ro(A'^)-equivalence classes represented by 
elements in A.k,d,Qi is disjoint from the set of ro(A/")-equivalence classes represented 
by elements in Ak^d,g2- Now, we want to count the number of non-empty sets Ak,d,e- 
That is, we want to count the number of solutions g of the equation 



(4.8) g^ = d (mod4/c), 1 ^ ^ ^ 2A;. 

If {d, fc) = 1, by Theorem 3.4 of Chapter 12, Hua [4] the number of solution of 
the equation (4.8) is equal to 



p\k 



n 1+ 



Next, we consider the case when there exists a prime number q satisfying q\k and 
q'^\d. Then we have q\g, and the equation (4.8) can be written as 

/A f q\'^ d , . 4k. ^ g ^ 2k 

(4.9 [^] =^ (mod — , 1< ^ ^ — . 

\qj q^ q q q 

Dividing out g,d and k by all such prime numbers q as in (4.9), we can reduce 
the second case to the first case when {d,k) = 1. Then, by using properties of the 
Legendre symbol we obtain that the number of solution of the equation (4.8) in the 
second case is still equal to 



n 

p\k 



P 



Finally, we consider the case when there exists a prime number q satisfying q\k,q\d 
and q'^ f d. We have again q\g. The equation (4.8) can be written as 

, , f g\ d , 4k. p 2k 

(4.10) g - = - mod — ), 1 ^ - ^ — . 

\qj q q q q 
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We can assume that q ^ 2. Otherwise, if g = 2 then we must have q^\d. Then we 
have {q,4:k/q) = 1, and hence a number Xq exists such that qxq = 1 (mod 4:k/q). 
Then the equation (4.10) can be written as 

/ . N f d , Ak^ p 2k 

(4.11) - =Xq- mod — ), 1 ^ - ^ — . 

\qj q q q q 



Note that we have 
(4.12) 



Xqd/q\ f q\ f d/q\ ( d 



P J WJ \ V J \V 



for any prime number 'p\{klq). Dividing out all such primes q as in (4.11) and using 
(4.12), we obtain that the number of solution of the equation (4.8) in the final case 
is equal to 

n(-(f 

This completes the proof of the lemma. □ 

By Lemma 4.5, we get the following corollary. 

Corollary. Let he the number of indefinite primitive quadratic forms of dis- 
criminant d, which are not equivalent under Tq{N). Then we have 



1.1 AT „l U ^ ' ' 



A;|7V p\k 

Theorem 4.6. Let N he a square free positive integer with {n, N) = 1. Then we 
have 

{p}: Per*,rp#{i2} 

^ ^ m\Nk\N ^ ' ^ den u ^ / "Vu \ 

for 3?s > 1, where the summation on u is taken over all the positive integers u such 
that V4n + dk^u^ e Z. 

Proof. Let [oj, 6-,-, Cj], j = 1,2, ••• , Hd, be a set of representatives for classes of 
primitive indefinite quadratic forms of discriminant d, which are not equivalent 
under ro(A'"). By Lemma 4.4, we have 

{p}: Per*,rp/{i2} ^ ^ denj=i „ "V'^i \ / 
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for 3?s > 1, where di = dN'^{aj, N)~'^ and where the summatfon on u is taken over 
all the positive integers u such that \/An + div? e Z. Denote k = N/{aj, N). Then 
k\N. By using Lemma 4.5, we can write the above identity as 

E ^(^) 

{p}: Per*,rp^{i2} 

^ k\N d€Q u p^N \ \^/ / V i \ / 

for 3?s > 1, where di = dk'^. By using Dirichlet's class number formula 
and by using the identity (See Theorem 11.2 of Chapter 12, Hua [4]) 



p\k 

we can write (4.13) as 

E c(P) = iV^^-^ 

{p}: Per*,rp^{i2} ^ ^ 



X 



for 3?s > 1, where the summation on u is taken over all the positive integers u such 
that VlrT+dPt? e Z. Since 

nO-©)n(-(^).i=E(^)tf^. 



we have 



(4.14) 



{p}: P6r*,rp^{i2} ^ ' 



m\N k\N ^ ' 

Next, we want to show that 



(m,k) X"^ J u\fd \ 4n 

m\Nk\N ^ ' ' den u ^ ^ "VW \ 



2„,2\ 2-« 



den w 



TO/ V 4n 
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is absolutely convergent for cr = 3?s > 1. Since 
(4.15) 

-—s - - 

It is proved in Li [8] that the right side of (4.15) is convergent for cr > 1, and hence, 
the right side of the stated identity is absolutely convergent for > 1. 
This completes the proof of the theorem. □ 



Proof of Theorem 1. It is proved at the end of Li [8] that 



E 



y/du 4n 



hd In ed 



{du^y 



« E 

d€Q,u 



(du^) 



2\^+e-l-a 



U 



l+2e 



< OO 



for a > 0. Then it follows from Theorem 4.6 that 

1 



(4.16) 



lim (s 

s^l/2+iK 2 



{P},rp^{i2} 



(47r)i/2r( 



in] 



X 



E E W - -\-) E E (3 

m\Nk\N p\k d€Q u ^ ^ 



(4n)-V2-i«r(l/2 + m) 

d \ hdlned 

A] 

den u 



(du^) 



2\s ■ 



Theorem 3.12 shows that the function on the right side of (4.16) represents an 
analytic function in the half-plane > except for a possible pole at s = 1 /2 and 
for possible simple poles at s = 1, | ± inj, j = 1, 2, • • • . The stated identity then 
follows from Lemma 4.1. 

This completes the proof of the theorem. 
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